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METHOD FOR IMAGE PROCESSING 



BACKGROUND OF THE INVENTION 
5 1. Field of the Invention 

The present invention relates to an image 
processing method capable of properly recognizing the 
road condition ahead, especially a vehicle or like object 
ahead, by processing an image captured by a camera 
10 mounted on a vehicle. 

2. Description of the Related Art 

In the conventional art, it is practiced to 
recognize a vehicle, or like object, ahead or make range 
measurements, etc. using an image captured by a camera. 
15 In this case, to recognize the vehicle or like object 
ahead, edge extraction is performed to obtain an edge 
image and, based on the obtained edge image, the range 
and angle to the vehicle or like object ahead are 
measured. Refer, for example, to Japanese Unexamined 
20 Patent Publication No. 2001-101428. 

An edge is a contour line where the gray level 
of the image changes rapidly, and the advantage is that 
an edge remains relatively unaffected if the gray level 
changes due to a shadow or stain. In edge extraction, 
25 either vertical or horizontal edges, or both vertical and 

horizontal edges, can be extracted. 

When performing edge detection in image 
processing, either vertical or horizontal edges, or both 
vertical and horizontal edges, are detected using, for 
30 example, cameras arranged on the left and the right, and 

the range and angle to the target object are measured. 
However, depending on camera noise or surrounding 
conditions, edges may not appear at constant positions 
and, when detecting edges at predetermined intervals of 
35 time, it is often not possible to detect edges 

consistently. For example, an edge of a particular 
portion of a vehicle traveling ahead may not appear at a 
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certain instant in time or may not appear at a constant 
position, making it impossible to consistently detect the 
edge of that portion. 

It is accordingly an object of the present 
5 invention to provide an image processing method that can 
compensate for such inconsistent edge detection. 

SUMMARY OF THE INVENTION 

According to the present invention, the field of 
10 view of a stereo camera is segmented in the form of a 
matrix by angle and by measured range value based on 
parallax, and matrix data is calculated for a segment 
where a detected edge exists, and also for segments 
surrounding the segment, the calculated matrix data then 
15 being assigned to each of the segments; then, a search is 
made through the assigned matrix data to find a segment 
that has matrix data exceeding a predetermined threshold 
value, and if the detected edge exists in the thus found 
segment, edge data of the edge is taken to represent the 
20 position of an object. 

If the edge does not exist in the thus found 
segment, then the threshold value is successively 
increased, a search is made successively to find a 

segment that has matrix data exceeding the increased 

/ 

25 threshold value and, if an edge detected at a past poxnt 
in time corresponding to the increased threshold value 
exists in the thus found segment, edge data of the edge 
is taken to represent the position of an object. 

As the matrix data is used in addition to the 

30 detected edge data, as described above, the present 

invention can reliably detect the position (range and 
angle) of the object. Further, even when the edge 
detected at a particular position is not detected at a 
certain instant in time, the position (range and angle) 

35 of the object can be consistently detected by using the 

matrix data. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The above object and features of the present 

invention will be more apparent from the following 

description of the preferred embodiments with reference 
5 to the accompanying drawings, wherein: 

Fig. 1 is a block diagram showing the configuration 

of an image processing apparatus according to the present 

invention . 

Fig. 2 is a diagram showing an image captured by a 
10 camera. 

Fig. 3 is a diagram showing vertical edges extracted 
from the image captured by the camera. 

Fig. 4 is a diagram showing an image captured by the 
camera . 

15 Fig. 5 is a diagram showing vertical edges extracted 

from the image captured by the camera. 

Fig. 6 is a diagram showing horizontal edges 
extracted from the image captured by the camera. 

Fig. 7 is a diagram showing the principle of how the 
20 distance from the cameras to the target object is 
determined using stereoscopic images in the image 
processing apparatus shown in Fig. 1. 

Fig. 8 is a diagram showing the parallax versus 
measured range relationship for constructing a matrix to 
25 be used in the present invention. 

Fig. 9 is a diagram depicting the matrix to be used 
in the present invention. 

Fig. 10 is a diagram showing how matrix data is 
assigned to each matrix segment. 
3 0 Fig. 11 is a flowchart showing one embodiment of the 

present invention. 

Fig. 12 is a flowchart showing another embodiment of 
the present invention. 

Fig. 13 is a diagram showing how a coefficient is 
35 assigned to each matrix segment. 
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Fig. 1 is a block diagram showing the configuration 
of an image processing apparatus used to implement the 
image processing method of the present invention. The 
image processing apparatus 1 is equipped with a pair of 
5 left and right cameras L and R for capturing images in 

the path ahead, and measures the distance 
stereographically by using the images captured by the 
cameras L and R. More specifically, the distance to the 
target object in front, for example, a vehicle traveling 

10 ahead, is measured using the two cameras, and vehicle-to- 
vehicle distance control, etc* is performed to maintain a 
constant distance to the vehicle ahead. 

Image data obtained from the cameras L and R are 
amplified by respective amplifiers 11 and 12, and band- 

15 limited by respective filters; the image data are then 

input to respective A/D converters 13 and 14, and stored 
in a memory 15. The image data stored in the memory 15 
are processed by a CPU 16, and used by a vehicle-to- 
vehicle distance control unit (ECU) 3 for such purposes 

20 as vehicle-to-vehicle distance control to control the 
distance from the vehicle ahead. The CPU 16 processes 
the image data obtained from the cameras L and R, and 
obtains information such as the distance, detection 
angle, height, width, etc. of the object located ahead 

25 such as the vehicle traveling ahead. 

Fig. 2 is a diagram showing an image captured by a 
camera. Fig. 3 is a diagram showing vertical edges 
extracted from the image captured by the camera shown in 
Fig. 2. In Fig. 3, only the vertical edges of the 

30 vehicle ahead are depicted, but the vertical edges of 

lanes can also be extracted. 

Edge information is information generated by 
extracting points where the gray level of the image 
changes, while range (distance) information is 

35 information generated by comparing images captured by a 

multiocular camera and thereby obtaining the parallax for 
range measurement. Edge information can also be obtained 
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from an image captured by a monocular camera; in this 
case also, pattern recognition can be done, and the 
approximate distance to the target object and the 
position and angle of the object's position can be 
5 determined. 

Fig. 4 is a diagram showing an image captured by the 
camera when an overhead bridge 3 with a road sign 4 
attached thereto is located ahead- Fig. 5 is a diagram 
showing an edge image generated by extracting vertical 

10 edges from the image shown in Fig. 4. When the vertical 
edges are extracted, only lines showing the piers of the 
bridge 3 and the vertical sides of the road sign 4 
appear, as shown in the figure. Horizontal edges of the 
image can likewise be extracted. Fig. 6 is a diagram 

15 showing an edge image generated by extracting horizontal 

edges from the image shown in Fig. 4. 

Fig. 7 is a diagram showing the principle of how the 
distance from the cameras to the target object is 
determined using stereoscopic images in the image 

20 processing apparatus shown in Fig. 1. The left and right 
cameras L and R respectively comprise imaging devices 33L 
and 33R, each constructed from a sensor chip formed on a 
camera substrate 32L or 32R, respectively, and lenses 31L 
and 31R for focusing an image of a vehicle V ahead onto 

25 the respective imaging devices. Reference numerals 34L 
and 34R are the optical axes of the respective cameras L 
and R, extending parallel to each other, and D is the 
distance between the two cameras. 

Here, the axis extending in a horizontal direction 

30 and parallel to the plane of the imaging devices 33L and 

33R is denoted as X axis, the axis extending in a 
vertical direction and parallel to the plane of the 
imaging devices 33L and 33R is denoted as Y axis, the 
axis extending in the same direction as the optical axes 

35 is denoted as Z axis, and the position of the target 

object V is designated by p(x, y, z); then, the distance 
z from the cameras to the target object V can be 
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expressed by the following equation, 
z = f*D/(x L -x R ) = f*D/S 

In the above equation, f is the focal distance, and 
D is the distance between the two cameras. Further, x L - 
5 x R = S is the parallax, (x L , y L ) is the position of the 
point at which the position p is imaged on the image ML 
captured by the camera L, and (x R , y R ) is the position of 
the point at which the position p is imaged on the image 
MR captured by the camera R. When the horizontal axes of 

10 the imaging planes are aligned on the same line as shown 
in Fig. 7, the parallax S = x L -x R . The focal length f, 
the parallax S, and x L , x R are expressed in terms of the 
number of pixels, while the distance D between the two 
cameras and the distance (range) z are both expressed in 

15 meters. 

Fig. 8 is a diagram showing the relationship between 
the parallax and the measured distance (measured range) 
for constructing a matrix to be used in the present 
invention. The measured range value R is given as R = 

20 K/n. Here, K is a constant determined by the parameters 
of the stereo camera, and n is the parallax. Fig. 8 
shows the case of K = 140. The parallax n increases as 
the range to the object decreases. Accordingly, as the 
parallax value decreases, the measured range value R 

25 increases. 

The measured range value R thus changes as the 
parallax value changes but, in the present invention, the 
measured range value R for an integer value of the 
parallax n is taken as the measured range value R for a 

30 given range of parallax values centered about that 

integer value, and the measured range value R is plotted 
as a staircase pattern. For example, the measured range 
value 70 for the parallax n = 2 is taken to represent a 
set of measured range values for the parallax value range 

35 of 2.5 > n ^ 1.5. 

Fig. 9 is a diagram depicting the matrix to be used 
in the present invention. As shown, the field of view of 
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the camera CM spreads out radially. The field of view is 
divided into segments along the range direction in 
accordance with the measured range values for the 
respective integer values of the parallax n shown in Fig. 
5 8. For example, the farthest segment in Fig. 9 is the 
segment for n = 1, and the next farthest segment is the 
segment for n = 2. 

On the other hand, the angle of the field of view of 
the camera CM is divided into segments of equal angle 

10 (for example, 5°) and, based on the thus divided range 

and angle, the field of view of the camera is segmented 
in the form of a matrix and coordinates are assigned to 
each of these matrix segments. Fig. 9 shows an example 
of how coordinates are assigned to the respective 

15 segments. As shown, coordinates M(m, n), defined by the 
number m based on the divided angle (in the illustrated 
example, 1 to 9 ) and the range number n based on the 
parallax (in the illustrated example, 1 to 6), are 
assigned to each segment of the matrix. For example, the 

2 0 segment whose angle number is "5", and whose range number 

based on the parallax is " 1" , is assigned the coordinates 
M(5, 1). 

In the present invention, matrix data is calculated 
and assigned to each matrix segment specified by the 
25 above-defined coordinates, and the matrix data is used as 
an index for determining the presence or absence of an 
object . 

Fig. 10 is a diagram showing an example in which 
matrix data is actually assigned to each matrix segment 

30 formed by segmenting the field of view of the camera. 

The matrix data represents, as a numeric value, the 
frequency with which an object's edge appears for each 
matrix segment. For example, when an object's edge is 
detected at position A at a certain instant in time, 100 

35 is assigned as matrix data to the matrix segment 

corresponding to that position, and 50 is assigned as 
matrix data to each of the matrix segments neighboring on 
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(Upper neighbor) Mxd(t)(m, n-1) = Mxd(t-l)(m, n-1) + 

(P/2) 

(Lower neighbor) Mxd(t)(m, n+1) = Mxd(t-l)(m, n+1) + 

(P/2) 

5 At the same time, the matrix data is calculated for 

each of the matrix segments neighboring on the diagonally 
upper and lower sides of the above matrix segment, and 
the matrix data obtained by the calculation are 
respectively assigned to the diagonally neighboring 
10 matrix segments (S4). 

For example, the matrix data for the matrix segment 
diagonally neighboring on the upper left side is 
calculated as shown below, 

(Upper left neighbor) Mxd(t)(m-1, n-1) = Mxd(t-l)(m- 
15 1, n-1) + (P/3) 

According to the example shown in Fig. 10, P/3 = 30. 
Similarly, the matrix data for the matrix segments 
diagonally neighboring on the upper right, lower left, 
and lower right sides, respectively, are calculated as 
2 0 shown below. 

(Upper right neighbor) Mxd(t)(m+1, n-1) = Mxd(t- 
l)(m+l, n-1) + (P/3) 

(Lower left neighbor) Mxd(t)(m-1, n+1) = Mxd(t-l)(m- 
1, n+1) + (P/3) 

2 5 (Lower right neighbor) Mxd(t)(m+1, n+1) = Mxd(t- 

l)(m+l, n+1) + (P/3) 

In the above example, P, P/2, and P/3 are added, but 

instead, prescribed matrix data PI, P2, and P3 (where PI 

> P2 > P3) may be added. If the object is not detected 
30 at a certain instant in time, predetermined matrix data 

is subtracted. 

Next, the matrix data of each matrix segment is 

checked to search for a matrix segment Mp0(m, n) that has 

matrix data exceeding a predetermined threshold value 
35 PthO (S5); then, it is determined whether the edge 

detected at time t exists in the segment Mp0(m, n) (S6). 

If it does, the edge data of that edge is taken to 
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represent the position of the object. The edge data here 
means the edge data Rp(t) and 8p(t) of the edge existing 
in the matrix segment Mp0(m, n), taken from among the 
range Rx(t) and angle 6x(t) of each edge detected at time 
5 t and stored in SI. Of the data detected in SI, the edge 

data Rp(t) and 0p(t) is the edge data of the edge 
existing in the segment whose matrix data exceeds the 
predetermined threshold value, and therefore, the data of 
the edge whose presence is consistently detected. 

10 Besides the edge data of the edges detected at time t, 

the data of the edges detected at times t-1, t-2, t- 
n are also stored. 

If it is determined in S6 that the edge exists in 
the segment Mp0(m, n) (Yes), its edge data Rp(t) and 

15 0p(t) are taken as the range and angle of the object. If 
the opposite is true in S6, the process is terminated, 
and the same flow is repeated. 

Fig. 12 is a flowchart illustrating the operation of 
another embodiment of the image processing method 

20 according to the present invention. The process up to S6 

is the same as that shown in Fig. 11. If the edge 
detected at time t exists in the segment Mp0(m, n) in S6, 
the process proceeds to S7, as in the embodiment of Fig. 
11. 

25 In the embodiment of Fig. 12, on the other hand, if 

the edge data detected at time t does not exist in the 
matrix segment Mp0(m, n) whose matrix data exceeds the 
predetermined threshold value PthO (No), the matrix data 
threshold value is increased, and a search is made to 

30 find a segment that exceeds the thus increased threshold 

value; then, it is determined whether the edge detected 
last time (at time t-1) exists in the thus found segment 
that exceeds the increased threshold value. If it does, 
the edge data of that edge is taken to represent the 

35 position, i.e., the range and angle, of the object. If 

the edge detected last time (at time t-1) does not exist 
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in the segment that exceeds the increased threshold 
value, the threshold value is further increased, and it 
is determined whether the edge detected two detection 
cycles back (at time t-2) exists. Thereafter, a similar 
5 process is repeated. 

The operation of this embodiment will be described 
below with reference to the flowchart of Fig. 12. In the 
embodiment shown in Fig. 12, if the answer in S6 is No, 
the matrix data of each matrix segment is checked to 

10 search for a matrix segment Mpl(m, n) that has matrix 

data exceeding a predetermined threshold value Pthl (Pthl 
> PthO) (S8); then, it is determined whether the edge 
detected at time t-1 exists in the segment Mpl(m, n) 
(S9). If it is determined that the edge detected at time 

15 t-1 exists (Yes), its edge data Rp(t-l) and 9p(t-l) are 

taken as the range and angle of the object (S10). If, in 
S9, the edge does not exist (No), the threshold value is 
increased to Pth2 (Pth2 > Pthl), and a similar process is 
repeated. If the edge does not exist, the threshold 

20 value is further increased. In Sn, the threshold value 

is increased to Pthn (Pthn > ... > Pthl > PthO), and a 
search is made for a matrix segment Mpn(m, n) that has 
matrix data exceeding the threshold value Pthn (Sn); 
then, it is determined whether the edge detected at time 

25 t-n exists in the segment Mpn(m, n) (Sn+1). If it is 
determined that the edge detected at time t-n exists 
(Yes), its edge data Rp(t-n) and 0p(t-n) are taken as the 
range and angle of the object (Sn+2). If No in Sn+1, the 
process is terminated. 

30 in the above embodiment, the process is repeated by 

successively increasing the threshold value Pth from PthO 
up to Pthn, but the number of repetitions can be set as 
desired . 

In the above description, the range and angle of the 
35 object are detected using a pair of cameras, but it is 

also possible to detect the position (range and angle) of 
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the object by using a monocular camera; in that case, a 
matrix is constructed based on the range to the object 
obtained from the upper and lower positions on the 
screen , and on the prescribed angle with which the field 
5 of view of the camera is divided, and the position of the 
object is detected using the matrix data in addition to 
the edge data. 

In the above embodiments, when an edge is detected, 
the coordinates of each matrix located within its range 

10 and angle are obtained, and the matrix data is added by 
calculation. However, it is also possible to obtain the 
matrix data, for example, by multiplying with a 
coefficient Q as in 

Mxd(t)(m, n) = Mxd(t-l)(m, n) x Q 

15 Fig. 13 is a diagram showing an example in which a 

coefficient is actually assigned to each matrix segment 
when obtaining the matrix, data by multiplication. For 
example, when an object's edge is detected at position A 
at a certain instant in time, the matrix data of the 

20 matrix segment corresponding to that position is 

multiplied by a coefficient 1.5. Likewise, the matrix 
data of each of the matrix segments neighboring on the 
left, right, upper, and lower sides is multiplied by a 
coefficient 1.3, while the matrix data of each of the 

25 matrix segments neighboring diagonally on the upper and 
lower sides is multiplied by a coefficient 1.1. As the 
calculation performed here is not an addition, the matrix 
data of each matrix segment is initially set, for 
example, to "1". Then, when an edge is detected, the 

30 matrix data of each matrix segment is multiplied by its 
corresponding coefficient shown in Fig. 13. If the 
object is not detected, each matrix data is multiplied, 
for example, by a coefficient 0.5. However, if the 
multiplication by 0.5 is repeated a number of times, the 

35 matrix data value will become so small that the value 

hardly increases even if it is multiplied by 1.5 when the 
object is detected; in view of this, when the matrix data 
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becomes smaller than 1, the matrix data value is 
increased to "1". When obtaining the matrix data by 
multiplication as in this example , the "predetermined 
threshold value" shown in Fig. 11 is set, for example, to 



